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FOREWORD 

The  National  Voluntary  Laboratory  Accreditation  Program  (NVLAP), 
established  in  1976,  is  administered  by  the  NBS  Office  of  Product  Standards 
Policy.  NVLAP  is  a voluntary  system  for  assessing  and  evaluating  testing 
laboratories  and  accrediting  those  found  competent  to  perform  specific  test 
methods  or  types  of  test  methods.  Individual  laboratory  accreditation 
programs  (LAPs)  for  specified  product  or  service  areas  are  established  in 
response  to  requests  and  demonstrated  need. 

This  publication  is  the  main  body  of  written  guidance  provided  to  NVLAP 
technical  experts  (TEs),  who  serve  as  assessors  and  evaluators.  The 
general  policies  and  practices  of  NVLAP  assessment  and  evaluation  are 
described.  The  specific,  technical  details  for  assessing  and  evaluating 
laboratories  under  each  LAP  are  provided  in  appendices  tailored  for  each 
LAP.  The  appendices  are  not  part  of  this  publication. 
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I.  INTRODUCTION 


Purpose 

This  manual  is  for  use  by  NVLAP  technical  experts  (TEs).  It  describes  the 
role  of  a NVLAP  TE  in  performing  the  assessment  and  evaluation  functions  of 
NVLAP. 

The  information  presented  here  is  intended  to  spell  out  your 
responsibilities  as  a TE.  Since  you  will  be  representing  NVLAP  directly 
with  applicants,  you  should  understand  the  entire  NVLAP  accreditation 
process,  including  NVLAP  policies  and  procedures,  sequence  of  events,  as 
well  as  the  on-site  assessment.  The  NVLAP  accreditation  process  is 
described  in  handbooks  specially  prepared  for  each  laboratory  accreditation 
program  (LAP).  LAP  handbooks  are  part  of  the  application  package  provided 
to  potential  applicants.  In  addition  to  describing  the  accreditation 
process,  each  LAP  handbook  identifies  the  specific  criteria  and  other 
technical  requirements  for  accreditation. 


NVLAP  Approach  to  Accreditation 

NVLAP  is  a system  for  accrediting  testing  laboratories  found  competent  to 
perform  specific  testing  services.  Before  an  accreditation  decision  is 
made,  a thorough  evaluation  is  performed.  The  NVLAP  evaluation  consists  of 
a review  of  application  information,  periodic  on-site  assessments,  results 
of  required  proficiency  testing,  and  correspondence  between  the  applicant 
and  NVLAP.  The  goal  of  an  evaluation  is  to  verify  that  a laboratory  has 
attained  a minimum  level  of  competence  as  defined  by  the  conditions  and 
criteria  for  accreditation.  However,  we  also  encourage  laboratories  to 
improve  testing  performance  above  the  minimum  level  through  the  use  of 
proficiency  testing  and  by  using  technical  experts  as  assessors  and 
evaluators. 

The  NVLAP  approach  to  accreditation  is  basically  non-adversarial . We  view 
our  role  as  one  of  assisting  applicants  to  attain  and  maintain  their 
accredited  status.  However,  we  are  firm  in  requiring  compliance  with 
accreditation  requirements  and  a laboratory's  accreditation  may  be  denied, 
suspended  or  revoked  if  compliance  is  not  achieved. 


Role  of  NVLAP  Technical  Experts 

NVLAP  TEs  have  a dual  role  in  the  accreditation  process.  One  role  is  to 
conduct  on-site  assessments  of  applicants.  The  other  role  is  to  evaluate 
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information  on  applicants,  including  assessment  reports,  responses  to 
deficiencies,  and  proficiency  test  results,  for  the  purpose  of  recommending 
appropriate  accreditation  actions. 

Your  main  contribution  is  your  technical  expertise.  You  and  the  other  TEs 
assigned  to  your  LAP(s)  provide  the  technical  direction  and  opinions 
required  to  evaluate  each  applicant's  compliance  with  the  accreditation 
requirements.  NBS/NVLAP  staff  provide  the  structure  and  administrative 
procedures;  you  provide  the  technical  support.  NBS  technical  staff  and 
others  may  also  be  available  for  consultation. 

One  important  aspect  of  NVLAP  is  that  each  applicant  be  evaluated,  in 
relation  to  others,  as  uniformly  as  possible.  Assessments  and  evaluations 
must  be  fair  and  even-handed.  Interpretations  of  requirements  should  be 
consistent  among  all  TEs  within  a LAP.  If  you  are  in  doubt  about  a 
requirement,  please  feel  free  to  discuss  it  with  other  TEs  as  well  as  the 
relevant  NVLAP  staff. 


Assessment  Assignments 

We  assign  TEs  to  perform  assessments  by  matching  individual  expertise  with 
a laboratory's  requested  scope  of  accreditation,  as  well  as  by  trying  to 
avoid  any  conflict  of  interest.  If  you  are  assigned  to  assess  or  evaluate 
a laboratory  for  which  possible  conf lict-of-interest  questions  could  be 
raised,  please  advise  us.  Although  it  is  not  always  possible,  assignments 
are  scheduled  so  that  every  participant  gets  a different  assessor  on 
successive  visits. 

Laboratories  have  the  right  to  refuse  acceptance  of  an  assessor  assignment. 
Although  this  rarely  happens,  if  a laboratory  does  not  want  you  to  be  the 
assessor,  notify  NVLAP  staff  immediately.  When  we  have  before-the-fact 
information  about  whether  a laboratory  will  accept  a certain  TE,  we  contact 
the  laboratory  before  making  the  assignment. 

When  you  are  assigned  to  perform  an  assessment,  we  will  send  you  copies  of 
the  laboratory's  application  form,  proficiency  test  data,  and  any  other 
pertinent  information.  Please  keep  these  documents  in  confidence.  After 
completing  the  assessment,  you  should  return  the  documents  to  NVLAP. 

Normally,  you  will  have  at  least  60  days  to  schedule  and  complete  the 
assessment(s)  that  you  have  been  assigned.  Occasionally,  a faster  response 
may  be  requested. 

On  your  first  one  or  two  assessments  you  will  normally  be  accompanied  by  a 
NVLAP  staff  person.  After  this  initial  visit(s)  you  will  generally  be  on 
your  own.  Occasionally,  you  may  be  accompanied  by  a NVLAP  staff  person  or 
possibly  a new  TE  in  training. 
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Monitoring  Visits 


In  addition  to  regularly  scheduled  on-site  visits,  you  may  be  asked  to 
conduct  monitoring  visits  which  may  be  scheduled  for  cause  or  on  a random 
selection  basis  during  the  accreditation  period.  These  visits  serve  to 
verify  reported  changes  in  the  laboratory  facilities  and  operations  or  to 
explore  possible  reasons  for  poor  performance  in  proficiency  testing.  The 
scope  of  the  monitoring  visit  may  range  from  checking  a few  designated 
items  from  the  checklists  to  a complete  review.  An  assessment  report  is 
still  written  and  left  with  the  laboratory  as  is  done  during  a regular 
assessment,  but  not  all  checklist  items  need  be  covered  or  completed. 

When  we  make  assignments  for  monitoring  visits,  we  will  specify  what  should 
be  examined,  but  you  are  not  bound  by  our  instructions  if,  while  you  are  at 
the  laboratory,  you  find  other  items  that  need  to  be  reviewed.  Monitoring 
visits  generally  last  no  longer  than  one  day. 


II.  ON-SITE  ASSESSMENT 


The  on-site  assessment,  a key  part  of  the  accreditation  process,  is  our 
face-to-face  examination  of  the  laboratory.  Before  initial  accreditation, 
and  periodically  thereafter,  on-site  assessments  of  each  NVLAP  participant 
are  conducted  to  assess  compliance  with  accreditation  requirements. 
Assessments  may  last  from  one  to  several  days  and  may  involve  more  than  one 
assessor  depending  on  the  scope  of  accreditation  desired  by  the  laboratory. 

Specific  aspects  of  a NVLAP  on-site  assessment  are  described  under  separate 
headings  below. 


Assessment  Forms 


NVLAP  uses  on-site  assessment  forms  (checklists  and  an  assessment  report 
form)  to: 

o ensure  uniformity  of  assessments  among  assessors; 
o guide  each  assessment; 

o provide  a place  for  recording  notes  during  an  assessment; 

o help  assessors  prepare  exit  briefings  and  assessment  reports; 

o provide  evidence  that  an  assessment  was  performed  and  that  all 
applicable  aspects  of  the  laboratory  were  examined;  and 

o provide  information  for  future  assessments,  thus  facilitating 
continuity. 
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The  structure  and  content  of  assessment  forms  can  vary  depending  on  the 
LAP.  Each  assessment  form  contains  checklists  and  a report  form.  Specific 
instructions  are  provided  with  the  on-site  assessment  form  used  in  your 
LAP(s) . 

Some  general  tips  in  completing  the  checklists: 


o Make  sure  that  each  item  on  the  checklists  either  is  answered  or  that 
there  is  a definite  indication  that  no  answer  was  feasible  or 
applicable. 

o Keep  as  many  notes  as  possible  on  the  checklists;  where  there  is 
insufficient  space,  use  additional  sheets  of  paper;. 

o Although  many  of  the  items  on  the  checklists  are  arranged  in  a 
suggested  sequence,  you  may  alter  the  sequence  as  you  see  fit. 


Preparation  for  an  Assessment 

The  overriding  purpose  of  preparing  for  an  assessment  is  to  organize 
yourself  to  use  the  time  available  during  the  assessment  to  the  best 
possible  advantage.  Failure  to  adequately  prepare  will  slow  down  the 
assessment,  create  an  unfavorable  impression  of  NVLAP  and  yourself,  waste 
the  laboratory's  time,  and  render  the  results  less  valuable.  The  following 
steps  are  recommended  when  preparing  for  an  assessment: 

1.  Notify  the  laboratories  at  least  one  month  in  advance  whenever 
possible. 

2.  Discuss  with  each  laboratory,  by  telephone,  the  laboratory  location, 
directions,  dates,  time  of  arrival,  security  arrangements,  names  of 
laboratory  staff  you  will  meet,  suitable  lodging  arrangements, 
proposed  agenda,  requested  demonstrations  of  specific  tests,  etc.  Ask 
the  laboratory  to  send  you  appropriate  information  to  review  before 
the  visit  (e.g.,  QA  manual  and  procedures).  Assure  the  laboratory 
that  any  documents  they  are  willing  to  send  you  in  advance  will  be 
held  in  confidence,  will  be  returned  upon  completion  of  the 
assessment,  and  will  not  be  retained  in  NVLAP  files  nor  your  own 
personal  files. 

3.  Review  all  relevant  information  (e.g.,  criteria,  test  methods,  QA 
manual,  application  information,  previous  assessment  reports, 
deficiency  correction  letters,  proficiency  test  results).  Prepare 
your  own  questions  based  on  previous  assessment  reports  and 
correspondence  to  address  weaknesses  previously  observed.  Review  the 
laboratory's  proficiency  test  results  and  look  for  trends  that  might 
indicate  an  apparatus  or  procedural  weakness.  This  information  may 
indicate  what  tests  need  to  be  demonstrated. 

4.  Develop  an  agenda.  If  time  allows,  send  a proposed  agenda  to  the 
laboratory  beforehand  and  state  who  at  the  laboratory  you  would  like 
to  have  available,  particularly  to  attend  the  exit  briefing. 
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5.  Complete  what  you  can  on  the  assessment  forms  before  your 
assessment. 


Entry  Briefing 

Upon  arrival  at  a laboratory  you  are  going  to  assess,  conduct  an  entry 
briefing  during  which  you  should: 

o meet  with  the  laboratory's  authorized  representative  and  all  others 
who  will  be  involved  in  assisting  the  assessment; 

o explain  the  purpose  of  the  assessment  and  set  the  agenda; 

o ask  for  one  escort  who  is  knowledgeable  about  the  laboratory's 
management,  quality  assurance,  and  procedural  systems  to  ensure 
cooperation  of  laboratory  personnel; 

o confirm  that  any  laboratory  documents  that  you  may  have  previously 
reviewed  are  still  current; 

o request  a meeting  room  or  place  where  you  can  be  by  yourself  to 
review  laboratory  documentation  and  records,  to  complete  the 
checklists,  and  to  compose  your  findings; 

o discuss  the  schedule  for  the  day,  conforming  as  far  as  possible  to 
the  laboratory's  working  hours,  lunch  hour,  and  coffee  breaks; 

o agree  on  a tentative  time  for  holding  the  exit  briefing; 

o write  down  all  names  and  positions  of  relevant  persons  for  future 
reference; 

o note  who  speaks  for  the  laboratory  and  who  is  knowledgeable  so  that 
you  may  refer  back  to  the  appropriate  "information  resource"  later  in 
the  assessment;  and 

o walk  through  the  portion  of  the  site  to  be  assessed  to  see  the  layout 
and  to  meet  the  relevant  laboratory  personnel. 


Typical  Steps  of  an  Assessment 

After  the  entry  briefing,  an  assessment  may  be  performed  in  any  order.  To 
some  extent,  the  checklist  questions  follow  a logical  sequence  and  dictate 
the  order  to  pursue  for  some  aspects  of  the  assessment,  but  you  need  not 
follow  that  order.  Some  assessors  plan  a sequence  of  activities  with  which 
they  are  more  comfortable.  However,  keep  in  mind  that  the  goal  of  an 
assessment  is  to  verify  that  the  laboratory  complies  with  all  applicable 
NVLAP  accreditation  requirements.  A summary  of  a typical  sequence  of  an 
assessment  is  as  follows: 
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1.  Conduct  an  entry  briefing. 

2.  Examine  QA  manuals,  personnel  competency  records,  equipment 
maintenance  and  calibration  records,  sample  handling,  and 
recordkeeping  procedures. 

3.  Examine  test  plans,  reports,  data  logs,  and  related  records. 

4.  Trace  one  or  more  samples  from  receipt  to  final  test  report. 

5.  Witness  demonstrations  of  selected  procedures  by  laboratory  staff. 

6.  Interview  technicians. 

7.  Physically  examine  equipment  and  facilities. 

8.  Formulate  impressions  about  the  overall  management  and  organization. 

9.  Conduct  an  exit  bri ef ing--discuss  findings,  particularly 
deficiencies. 

10.  Complete  written  assessment  report,  have  laboratory  sign  it,  and 
leave  a copy. 


Preparing  an  Assessment  Report 

After  you  have  completed  the  checklists,  find  a place  where  you  can  review 
your  notes  on  the  checklists  and  collect  your  thoughts.  Summarize  your 
findings  on  the  assessment  report  form.  Write  anything  that  is  significant 
about  what  you  have  observed,  whether  complimentary  or  negative.  However, 
before  an  exit  briefing,  be  sure  to: 

o respond  to  all  applicable  checklist  items; 

o decide  which  findings  must  be  presented  as  deficiencies  requiring 
corrective  action  by  the  laboratory;  and 

o describe  each  deficiency  on  the  appopriate  assessment  report  sheet. 


Exit  Briefing 

At  a minimum,  the  laboratory's  QA  manager  or  equivalent  and  his  superior 
should  attend  the  exit  briefing.  It  is  useful  for  future  reference  to 
identify  the  attendees  at  the  exit  briefing  on  the  first  page  of  an 
assessment  report.  An  exit  briefing  will  usually  not  last  more  than  one 
hour. 

For  each  deficiency,  describe  the  finding,  the  justification  or  reason  for 
stating  the  finding  and,  if  appropriate,  the  proposed  corrective  action. 
Make  sure  that  the  laboratory  understands  each  deficiency  and  what  is 
required  to  achieve  compliance. 
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Some  of  the  deficiencies  you  identify  may  be  challenged.  Remember  that  the 
laboratory's  technical  staff  may  also  be  experts  and  differences  of  opinion 
may  arise.  Try  to  work  out  any  differences  and  be  satisfied  that  the  NVLAP 
criteria  have  been  met.  However,  don't  argue.  As  appropriate,  record  your 
finding  as  a deficiency  and  ask  the  laboratory  to  refute  it  in  their 
response  to  NBS.  The  NVLAP  accreditation  process  provides  a second  level 
evaluation  which  includes  a review  by  other  technical  experts  and  NVLAP 
staff  of  your  assessment  report,  the  checklists,  results  of  proficiency 
testing,  and  correspondence  from  the  laboratory  responding  to  the  cited 
deficiencies. 

Do  not  be  overly  concerned  about  differences  that  may  arise;  they  can  be 
worked  out  later.  Our  experience  to  date  indicates  that  participants  look 
to  NVLAP  assessors  for  guidance  and  do  not  consider  them  as  adversaries. 

Advise  the  laboratory  to  write  NBS  within  30  days  describing  action  taken 
or  planned  to  be  taken  for  each  specific  deficiency  identified  in  the 
assessment  report.  The  letter  must  be  signed  by  an  authorized  official  of 
the  laboratory  who  can  certify  that  the  statements  made  in  the  letter  are 
correct. 

If  any  deficiencies  are  noted  at  laboratories  which  are  currently 
accredited,  such  deficiencies  must  be  corrected  within  30  days  or  they  may 
face  suspension  or  revocation  of  accreditation.  When  out-of-calibration 
apparatus  is  cited,  explain  that  the  apparatus  should  not  be  used  until 
corrective  action  has  been  completed. 

During  the  exit  briefing,  watch  for  warning  signs  of  difficulties  in 
obtaining  compliance.  For  certain  difficult  assessments,  a private 
conference  with  top  management  may  be  necessary  or  a separate  follow-up 
note  to  NBS  should  be  considered  if  there  were  major  problems  at  the 
laboratory  that  you  could  not  be  openly  or  comfortably  discussed  with  the 
laboratory  management. 


Completing  an  Assessment  Report 

Make  any  necessary  revisions  or  additions  to  the  assessment  report  in  light 
of  the  exit  briefing.  At  least  one  laboratory  representati ve,  preferably 
the  authorized  representative,  must  sign  the  assessment  report  indicating 
that  it  has  been  discussed  during  the  exit  briefing.  Leave  a copy  of  the 
report  with  the  person  who  signs  it  or  allow  the  laboratory  to  photocopy 
the  report.  Do  not  leave  a copy  of  the  checklists  (because  of  the 
possibility  that  your  "rough  notes"  will  contradict  your  official 
assessment  report  to  the  laboratory). 

After  you  have  left  the  laboratory,  mail  the  original  copy  of  the 
checklists  along  with  the  signed  original  of  the  report  to  NBS  and  retain  a 
copy  of  all  sheets  in  case  the  originals  are  lost  in  the  mail  or  we  need 
you  to  refer  back  to  them  during  the  technical  evaluation.  After  an 
accreditation  decision  is  made  by  NBS,  you  should  return  all  documents 
pertaining  to  the  assessment  to  NBS.  Mail  all  documents  to: 
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NVLAP 

National  Bureau  of  Standards 
ADMIN  A531 

Gai thersburg,  MD  20899 


III.  ASSESSMENT  CONDUCT  AND  TECHNIQUES 


More  than  your  technical  expertise  will  be  required  to  conduct  an 
assessment  successfully.  There  are  certain  procedural  items,  ethical 
issues,  assessment  concepts,  and  questioning  techniques  to  bear  in  mind. 
These  are  covered  in  this  section. 

Conducting  an  Assessment 

To  conduct  an  assessment  properly,  the  following  guidelines  should  be 
followed: 

o Make  every  effort  to  disrupt  as  little  as  possible  the  normal 

operating  schedule  of  the  laboratory  while  conducting  an  assessment. 

o Be  determined,  decisive  and  direct.  (Once  there  is  enough  evidence  to 
form  the  basis  for  a sound  judgement,  there  is  no  point  in  going  over 
the  same  ground.) 

o Keep  the  assessment  moving;  be  aware  of  the  overall  progress  of  the 
assessment  so  as  to  avoid  wasting  time  on  trivia. 

o Be  honest  and  fair.  (Personal  dislikes/prejudices  must  not  interfere 
with  the  assessment.) 

o Be  independent.  (You  decide  what  will  be  examined) 

o Recognize  the  difference  between  a "clean-up"  job  and  a "cover-up" 
job.  There  is  normally  no  need  to  worry  about  clean-ups  that  a 
laboratory  carries  out  before  you  arrive.  Clean-ups  rarely  solve 
major  problems,  but  show  that  the  laboratory  is  concerned  about  the 
outcome  of  the  assessment.  Dispense  with  cover-ups  and  get  to  the 
heart  of  each  problem. 

o Discuss  any  problems  right  away.  (This  deters  arguments  later  on  when 
people's  memories  have  grown  hazy,  and  allows  you  to  clarify  the 
problem  and  collect  new  information  relating  to  it.  If  the 
discussion  reaches  an  impasse,  do  not  get  involved  in  an  argument;  go 
on  to  the  next  item.) 

o Be  prepared  to  return  to  an  area,  if  necessary,  to  obtain  new 
information  and  to  reassess  the  operation  in  the  light  of  new 
information. 
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o Keep  a sense  of  proportion.  (The  magnitude  and  significance  of  a 
deficiency  is  a vital  issue.)  Make  allowances.  (Put  errors  in 
perspective.)  Don't  pursue  unimportant  errors,  which  wastes  time  and 
effort  and  risks  alienating  the  laboratory. 

o Be  constructive  at  all  times;  when  a deficiency  is  found,  suggest 
possible  corrective  action  if  you  are  certain  that  such  action  would 
satisfy  the  requirement.  If  wrong  in  your  assumptions  or 
conclusions,  it  is  best  to  admit  it  and  apologize.  Such  an  attitude 
elicits  cooperation  and  respect  from  the  laboratory. 

o Be  helpful;  Suggest  any  procedures  which  would  be  beneficial  if  the 
opportunity  presents  itself.  Your  helpful  suggestions  need  not  be 
NVLAP-oriented.  Do  what  is  reasonable  to  establish  an  attitude  that 
leads  the  laboratory  to  eagerly  welcome  the  next  assessor.  Be 
careful,  however,  that  you  do  not  give  away  some  other  laboratory's 
"trade  secret"  in  an  attempt  to  be  helpful.  Open  literature  sources 
are  preferred. 

o At  the  end  of  each  day,  give  the  laboratory  representatives  a brief 
summary  of  what  has  been  happening.  The  summary  is  a courtesy 
measure.  It  limits  the  need  to  make  embarrassing  revelations  at  the 
exit  briefing  and  in  the  assessment  report  by  forewarning  the  people 
of  the  nature  of  any  deficiencies  encountered.  In  some  situations 
the  deficiency  you  observe  early  in  your  assessment  can  be  corrected 
before  the  exit  briefing.  Most  laboratories  are  thankful  to  be  given 
the  chance  to  put  their  house  in  order  discreetly. 

o Try  to  answer  all  questions  posed;  if  you  lack  information,  offer  to 
provide  an  answer  at  a later  date  and  be  sure  to  follow  up,  or  refer 
questions  to  the  NVLAP  staff. 

o Thank  the  laboratory  for  its  assistance  and  hospitality.  Even  if 
there  were  some  contentious  issues  and  you  had  to  differ,  you  should 
still  indicate  that  you  appreciated  the  laboratory's  cooperation. 

o Treat  privileged  information  as  such. 

o Avoid  all  situations  which  could  be  construed  as  undue  or  improper 
influences  on  your  assessment  findings. 

o Do  not  request  favors  from  laboratories  which  could  be  construed  as 
improper  or  an  imposition. 

o Plan  to  pay  for  all  meals.  You  will  be  reimbursed  by  NBS.  Be 

prepared  to  eat  lunch  by  yourself  at  all  times,  but  if  the  laboratory 
staff  wishes  to  join  you,  that  is  acceptable.  In  some  cases  the  only 
reasonable  facility  is  a laboratory-supplied  eating  facility.  Lunch 
supplied  by  the  laboratory  in  these  cases  is  acceptable.  However, 
always  offer  to  pay. 

o Avoid  making  derogatory  remarks  about  specific  manufacturers  or 
suppliers  of  equipment  or  about  their  products. 
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o Avoid  making  derogatory  remarks  about  individuals  either  within  or 
outside  of  the  laboratory. 

o Avoid  becoming  involved  in  intra-laboratory  personnel  problems. 


Information  Collection 


Assessment  is  an  information  collection  process  that  entails  verifying 
application  information,  questioning  staff,  examining  facilities, 
equipment,  and  records,  and  reviewing  applicable  proficiency  testing 
results.  Collect  information  as  you  go  by  observing  and  listening 
(laboratory  staff  should  be  doing  the  majority  of  the  talking)  to  what  is 
going  on  around  you.  To  assist  in  completing  the  checklists,  it  may  be 
helpful  to  note  the  following: 

o key  persons  interviewed; 

o applicable  document  designations,  revision  dates,  where  found,  and 
descriptions; 

o equipment  numbers  and  identification  to  cross  reference  with  cali- 
bration and  maintenance  records; 

o sample/specimen  identification  system. 

o identification  of  document/information/equipment  recipients  in 
laboratory; 

o flow  charts  showing  how  the  laboratory  functions  in  terms  of  input 
(sample  receipt),  prerequisites  (people,  equipment,  etc.),  processing 
(testing  procedures),  and  output  (test  report).  (This  systems 
approach  to  viewing  a laboratory's  operation  may  be  useful  in 
identifying  weaknesses  and  underlying  causes  of  deficiencies);  and 

o latest  revision  dates  in  the  QA  manual,  procedures,  and 
instructions . 

It  may  also  be  useful  as  you  go  along  to  take  mental  notes  concerning  the 
following  questions: 

o Do  the  laboratory  staff  members  know  their  jobs? 

o Do  the  lab  managers/supervisors  want  to  answer  all  questions?  Some 
want  to  cover  all  questions  either  in  the  fear  that  their 
subordinates  will  commit  an  error  that  will  reflect  adversely  on 
them,  or  because  they  feel  it  is  their  right  and  responsibility  to 
speak  up  since  they  know  what  is  going  on.  In  this  situation,  you 
must  make  it  clear  that  the  "worker"  should  be  the  one  to  answer  the 
questions,  explaining  that  this  is  the  purpose  of  the  assessment  to 
see  what  is  actually  being  done  and  to  verify  that  the  proper 
procedures  are  being  implemented. 
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o Does  the  staff  try  to  bluff  its  way  out  of  a tight  corner? 

o Does  the  staff  know  the  QA  manual  and  associated  procedures  and  where 
these  documents  are? 

o What  is  the  condition  of  facilities  and  equipment?  Are  the  requisite 
calibration  tags  or  stickers  on  the  equipment? 

o Does  it  appear  that  the  QA  and  procedural  documents  are  used? 

Every  assessment  involves  collecting  information  on  a laboratory's  quality 
assurance  system.  The  objective  of  this  "information  collection"  is  to 
ascertain  whether  the  laboratory  can  recall  the  necessary  facts  to 
substantiate  its  test  reports.  Three  approaches  for  assessing  a 
laboratory's  quality  assurance  system  are  to: 

1.  trace  a test  through  the  testing  process  forward  from  receipt  of 
sample  to  final  test  report; 

2.  trace  a test  back  from  final  test  report  to  receipt  of  sample;  or 

3.  randomly  select  certain  files,  data  sheets,  reports,  etc. 

The  first  two  approaches  are  generally  more  efficient  and  effective.  Some 
questions  on  the  checklists  are  sequenced  to  use  these  two  approaches. 
Random  selection  may  be  too  haphazard  and  not  give  you  a complete 
understanding  of  the  system.  We  recommend  that  either  of  the  first  two 
approaches  form  the  baseline  investigation.  The  third  approach  is  used 
when  many  additional  test  methods  must  be  covered.  Keeping  careful  notes 
is  essential  to  help  you  prepare  a complete  assessment  report. 


Questioning  Techniques 

Effective  questioning  is  vital  to  an  assessment's  success.  It  is  of  great 
value  in  assessing  to  keep  the  systems  concept  in  mind,  to  separate  the 
laboratory's  work  into  input,  prerequisites,  process,  and  output,  and  to 
ask  questions  on  each  of  these  individually  and  in  turn.  You  should  voice 
your  understanding  of  how  the  system  works  so  that  the  laboratory  can 
correct  any  misconceptions. 

Ask  questions  that  require  a substantive  response  ( e . g . , "Who,  What,  How, 
When,  Where,  and  Why"  questions  force  a substantive  response)  rather  than  a 
"Yes/No"  answer.  Questions  that  require  a "Yes/No"  answer  are  useful  when 
you  need  to  clarify  particular  points.  Statements  or  questions  such  as: 
Show  me  what  you  do./  Where  do  you  put  that?/  How  do  you  do  that?/  are 
recommended.  If  there  is  a variation  from  written  procedures,  then  ask 
"Why  do  you  do  it  that  way?" 

Hypothetical  questions  are  good  when  little  objective  evidence  is 
available.  The  "What  if  . . ."or  "Let  us  suppose  . . ."  or  "I  don't 
understand  . . ."  type  of  question  is  usually  effective  in  such  situations. 
Avoid  asking  questions  that  suggest  the  answers  you  expect. 
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Silence  can  be  extremely  powerful.  Some  people  find  silence  uncomfortable 
and  to  break  it  they  may  volunteer  something  they  "shouldn't."  When  you 
look  at  people  and  say  nothing,  they  may  feel  a response  is  expected,  or 
because  they  are  not  quite  sure  what  the  response  should  be,  they  may  say 
more  than  they  otherwise  would.  This  often  produces  information  of  use. 

Observe  what  is  not  said  by  the  laboratory  staff.  If  answers  are 
superficial  or  evasive,  be  prepared  to  continue  the  line  of  questioning. 

Be  alert  to  differences  in  information  presented  to  you  from  different 
sources  ( i - e . , conflicting  answers  from  staff  members  or  differences 
between  documented  procedures  and  what  is  actually  observed  or  said). 

Attempt  to  put  the  staff  at  ease  at  the  beginning  of  a questioning  session 
since  they  will  probably  be  nervous  and,  if  so,  react  by  becoming  withdrawn 
or  defensive,  or  worse--by  responding  aggressively.  You  should  not  be 
aggressive  in  return,  but  you  must  be  thorough  (and  occasionally 
persistent)  enough  to  establish  the  factual  situation.  Keep  the  following 
points  in  mind: 

o Direct  questions  to  the  person  who  performs  the  task  being  assessed, 
and  not  to  that  person's  superior. 

o Never  talk  down  to  anyone. 

o Talk  the  "lab's  language". 

o Give  credit  where  credit  is  due.  A compliment,  sincerely  given,  goes 
a long  way  towards  eliciting  cooperation. 

o Be  interested  in  the  laboratory's  work  and  responses. 

o Don't  appear  to  be  distrustful  of  people  or  to  regard  their  responses 
with  criticism. 

o Be  calm  and  courteous  and  thank  people  for  their  time. 


IV.  DEFICIENCIES 
Identifying  Deficiencies 

A deficiency  is  a departure  from  or  an  instance  of  noncompliance  with  a 
condition  or  criterion  for  accreditation.  Keep  in  perspective  the 
magnitude  of  any  deficiency  found  and  avoid  making  an  issue  out  of  trivial 
deficiencies,  human  errors,  or  isolated  mistakes.  Isolated  errors  may  be 
due  to  someone  having  an  off-day.  Examine  records  for  clarity, 
completeness,  and  consistency,  and  be  alert  to  anomalies  which  may  be  a 
symptom  of  much  deeper  problems.  If  you  find  no  deficiencies  after  tracing 
the  testing  process  by  a random  selection  of  tests,  then  you  can  probably 
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conclude  that  there  is  a system  present,  working  and  known  to  the  personnel 
concerned.  If  only  one  or  a few  minor  deficiencies  are  found,  then  the 
system,  rather  than  not  being  implemented,  may  be  in  a period  of 
development  or  may  need  to  be  better  instilled  into  the  staff.  If  several 
deficiencies  are  found,  then: 

o the  system  may  need  to  be  revised  or  changed; 

o the  system  may  need  to  be  more  fully  developed  and  documented; 

o the  staff  members  may  need  to  be  more  adequately  trained  to  perform 
their  functions  as  part  of  the  system;  or 

o a combination  of  some  or  all  of  these  corrective  actions  need  to  take 
place. 

Make  sure  that  observed  deficiencies  are  discussed.  Sometimes  further 
information  can  be  provided  and  change  your  first  impressions.  When  a 
deficiency  is  found,  ask  questions  to  discover  its  cause.  After 
determining  the  underlying  cause,  you  should  try  to  suggest  corrective 
action  that  would  satisfy  the  requirements.  Always  remember  that  cheap, 
quick,  simple,  but  effective  corrective  action  is  more  likely  to  be  acted 
upon  promptly  by  the  laboratory,  particularly  if  it  sounds  easy  to 
impl ement . 

It  is  helpful  to  note  whether  a laboratory  is  already  aware  of  a 
deficiency.  If  so,  determine  what  action  has  been  taken  to  analyze  the 
cause  of  the  deficiency  and  to  implement  corrective  action.  If  the 
laboratory  was  unaware  of  the  deficiency,  encourage  the  laboratory  to  make 
a proposal  for  corrective  action. 

Obviously,  some  deficiencies  are  more  critical  than  others.  It  is 
virtually  impossible,  however,  to  provide  decision  rules  for  objectively 
ranking  deficiencies.  This  is  where  your  expert  technical  judgment  rules. 

The  causes  of  deficiencies  can  be  categorized  as  follows: 

o Lack  of  system-measures  have  not  been  taken  to  ensure  that  an 
activity  is  performed  or  performed  properly. 

o Lack  of  indoctrination/training  of  the  personnel --system  has  been 
developed  and  documented,  but  staff  is  unaware  or  doesn't  know  how  to 
use  it. 

o Lack  of  discipline  on  the  part  of  the  personnel  involved  in 

implementation  of  the  system  as  documented--the  personnel  have  been 
trained  in  how  to  operate  the  system  but  don't  comply. 

o Lack  of  time--too  much  pressure  of  work,  overwork,  or  inadequate 
manpower. 

o Lack  of  resources—incorrect  equipment, 
o Lack  of  top  management  support. 
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Typical  Deficiencies 


Deficiencies  are  usually  found  in  a laboratory's  documentation,  equipment, 
or  personnel.  Typical  documentation  deficiencies  include: 

o Required  items  in  the  criteria  are  not  addressed. 

o Contents  of  instructions  are  not  adequate. 

o Procedures/test  plans  do  not  have  sufficient  detail. 

o Outdated  copies  of  documents  are  still  in  use. 

o The  system,  as  described  in  the  QA  manual  or  instructions,  has  been 
ignored. 

o Samples/specimens  bear  no  identification  to  permit  traceability. 

o Responsibility  for  keeping  documents  updated  has  not  been  assigned. 

o No  planned  maintenance  records  of  equipment  are  available. 

Typical  equipment  deficiencies  include: 

o Equipment  is  out-of-tolerance  and/or  has  not  been  calibrated  or 
verified  at  requisite  intervals. 

o Cleanliness  is  inadequate. 

o No  provision  is  made  for  protected  storage  in  the  laboratory. 

o Substitute  equipment  is  used  without  acceptable  evidence  that  it 
performs  as  specified  in  the  standard  involved. 

Typical  personnel  deficiencies  include: 

o Supervision  is  inadequate. 

o On-the-job  practice  is  careless. 

o Personnel  are  not  adequately  trained  to  perform  their  tasks. 

o Objective  evidence  is  not  available  in  laboratory  files  that  attests 
to  the  training/certif ication  of  competence  of  the  personnel. 
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V.  TECHNICAL  EVALUATION 


Technical  evaluations  (second  level  reviews)  of  all  applicants  are  normally 
conducted  at  NBS.  The  usual  procedure  for  new  applicants  is  to  have  one  or 
more  TEs  and  NVLAP  staff,  who  were  not  involved  in  the  assessment,  review 
the  NVLAP  evaluation  folder  for  each  laboratory.  The  evaluators  provide 
technical  judgments  on  whether  an  applicant  complies  with  the  conditions 
and  criteria  for  accreditation  and  make  recomnendations  on  appropriate 
accreditation  actions  to  take  accordingly.  In  some  cases  a special  form  is 
used  to  record  TE  evaluation  recommendations.  A TE  recommendation  is  based 
on: 

o information  provided  on  the  application; 

o assessment  reports; 

o actions  taken  by  the  laboratory  to  correct  deficiencies  observed 
during  the  on-site  visit; 

o results  of  proficiency  testing  from  current  and  previous  assessment 
periods;  and 

o information  from  any  monitoring  visits  that  may  have  been  done. 

When  a laboratory  is  not  considered  to  be  in  compliance  with  all 
requirements,  we  notify  it  and  request  additional  information  to  allow  the 
laboratory  to  demonstrate  compliance.  Occasionally,  an  additional 
assessment  may  be  necessary. 

The  NVLAP  staff  prepares  the  paperwork  including  the  transmittal  of 
additional  correspondence  to  any  laboratory  which  the  evaluators  deem  needs 
to  respond  more  fully  to  unresolved  deficiencies  before  accreditation  can 
be  recommended. 

When  notification  describing  specific  deficiencies  is  made,  the  laboratory 
must  respond  within  30  days  of  notification  and  provide  documentation  or 
certification  by  an  authorized  member  of  management  that  the  specified 
deficiencies  have  been  corrected.  Clarification  of  some  issues  may  be 
requested  by  telephone.  All  deficiencies  must  be  corrected  before 
accreditation  is  granted. 

When  a laboratory  satisfies  all  requirements,  the  Director  of  the  NBS 
Office  of  Product  Standards  Policy  signs  and  transmits  a certificate  and 
scope  of  accreditation  to  the  laboratory. 
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